Meta unveils SAM Audio, a unified multimodal AI model to separate sounds using text or visual prompts

 Meta Unveils SAM Audio: Revolutionizing Audio Editing with Multimodal AI

SAM Audio represents Meta's latest breakthrough in the Segment Anything Model (SAM) family, enabling precise sound isolation from complex audio mixtures using text, visual, or temporal prompts. This open-source model automates tasks that once demanded manual expertise in tools like Adobe Audition.

Core Features

SAM Audio supports three intuitive prompting methods, which can combine for enhanced accuracy. Text prompts extract sounds like "dog barking" from podcasts; visual prompts let users click video objects, such as a guitarist, to isolate their audio; and span prompts mark time segments to remove recurring noises across files. It processes audio faster than real-time (RTF ≈ 0.7) across models from 500M to 3B parameters, handling speech, music, and environmental sounds.

Technical Architecture

The model relies on separate encoders for audio mixtures, text, visual cues from video masks, and time spans, feeding into a diffusion transformer for separation. Its Perceptual Encoder Audio-Visual (PE-AV) aligns video and audio features, enabling "hear with your eyes" capabilities even for off-screen events. Outputs include a "target" waveform for the isolated sound and a "residual" for the rest, streamlining edits like noise removal or stem extraction.

Availability and Access

Download SAM Audio via GitHub, Hugging Face, or Meta's site under a permissive SAM License for research and commercial use. Test it instantly in the Segment Anything Playground with personal audio/video files—no local setup required.

Real-World Applications

Content creators can clean podcast noise, musicians isolate stems for remixing, and filmmakers enhance post-production workflows. It excels in accessibility (e.g., filtering distractions), scientific analysis, and gaming audio tweaks, outperforming specialized tools in diverse scenarios. For developers like those building AI workflows, its open-source nature accelerates integration into apps.

SAM Audio challenges traditional audio software by democratizing pro-level editing, positioning Meta as a leader in multimodal AI for creators targeting YouTube Shorts or Instagram Reels. Early benchmarks show state-of-the-art results, with potential for your WebTechPoint channel to demo audio enhancements in tech tutorials.